To give a bit more context. Some of the numbers I was seeing are pasted below along with the queries ran against prometheus. The methodology was to create two 4.9 SNO-DU clusters (leader-election-new and leader-election-oc), both had CVO scaled down, and the only modification was that leader-election-new was running 11 operators that were rebuilt with the modified library-go running with the new leader-election numbers. I was seeing some small reductions, so I figured if we extrapolate that to the rest of the core operators we would see better gains in SNO-DU clusters, where we have very limited resources (currently management workloads are pinned to 2 cores).

Things I tried to verify was the downtime support for APIServer in SNO clusters described in this bz, rolling out APIServer didn't cause any restarts on the operators. Similarly rolling out the operator didn't cause any restarts and the operands behaved correctly with out any restarts as well.

Queries Ran

95th Quantile Over Time (CPU Usage) - API Server

quantile_over_time(
  .95,
  sum(
    rate(container_cpu_usage_seconds_total{
      container!="POD",
      pod=~"kube-apiserver-leader-election-oc-49|kube-apiserver-leader-election-new-49",
      namespace="openshift-kube-apiserver"}[5m]
    )
  )[6h:]
) * 1000

Average Over Time (CPU Usage) - API Server

avg_over_time(
  sum(
    rate(container_cpu_usage_seconds_total{
      container!="POD",
      pod=~"kube-apiserver-leader-election-oc-49|kube-apiserver-leader-election-new-49",
      namespace="openshift-kube-apiserver"}[5m]
    )
  )[6h:]
) * 1000

95th Quantile Over Time (CPU Usage) - Operators

quantile_over_time(
  .95,
  sum(
    rate(container_cpu_usage_seconds_total{
      container!="POD",
      namespace=~"openshift-apiserver-operator|openshift-authentication-operator|openshift-cluster-storage-operator|openshift-console-operator|openshift-controller-manager-operator|openshift-etcd-operator|openshift-kube-apiserver-operator|openshift-kube-controller-manager-operator|openshift-kube-scheduler-operator|openshift-kube-storage-version-migrator-operator|openshift-service-ca-operator"
      }[5m]
    )
  )[6h:]
) * 1000

Average Over Time (CPU Usage) - Operators

avg_over_time(
  sum(
    rate(container_cpu_usage_seconds_total{
      container!="POD",
      namespace=~"openshift-apiserver-operator|openshift-authentication-operator|openshift-cluster-storage-operator|openshift-console-operator|openshift-controller-manager-operator|openshift-etcd-operator|openshift-kube-apiserver-operator|openshift-kube-controller-manager-operator|openshift-kube-scheduler-operator|openshift-kube-storage-version-migrator-operator|openshift-service-ca-operator"
      }[5m]
    )
  )[6h:]
) * 1000

Summarizing the overall benefit to an SNO cluster over X timeframe will be helpful for weighing the tradeoff of slightly lower CPU cost at the expense of latency of lease reacquisition.

Sure thing, a quick summary with some new numbers. I ran the clusters under load. I used kube-burner to spin up some traffic against the API Server and just have some workloads in the cluster. The latest numbers show a pretty consistent picture of a small reduction in CPU consumption by the Changed Operators. On average by increasing leader election to 60s I'm seeing a reduction of 1-2 millicores per operator, extrapolating that to the 30 operators we run we could see 30millicore reduction on on average on the low end and 60millicore on the high end. That is a very small reduction but the gain is meaningful in environments where there is significant reduction in resources. Some DU configurations for example are pinning management workloads to 2 cores. This gives them very little breathing room and any measurable reduction is useful to provide more resources for Crio and Kubelet to operate. This does mean we reduce the responsiveness of our operators during rollouts, but the gain would mean that if need be we can support another operator in those limited environments.

In short, with the current numbers, we would have a worst non-graceful case of 330s before the new operator takes leader and a best graceful case of 60s which would mean a latency in lease reacquisition. The upside is that this would afford resource starved deployments some more breathing room in SNO environments (all be it small), which would offer Kubelet and Crio more room or potentially allow running another operator.

As always with things of this nature, another pair of eyes would be fantastic on the numbers, queries and the methodology. Any input is greatly appreciated.

/test unit

/approve

please squash before merge.

diff of diffs is clean https://github.com/openshift/library-go/compare/aa75fd24bdd8b2b1c798a1237bc27364a16bfac0..2612981f3019479805ac8448b997266fc07a236a

/lgtm

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, eggfoobar, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here